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Introduction 

On-line trial testing for fourth-grade students is an exploratory study realized as 
a part of the project "Developing annual test of students’ achievement in Nature 
& Society" realized by Institute for Education Quality and Evaluation [1]. Basic 
idea of the study was to enable students to participate in a large-scale testing 
from their own schools through the Internet. All schools where teachers were 
interested in the testing were eligible for participation in the study. Motivation 
for this mode of test delivery was the fact that the schools in Serbia have much 
better Internet connections today than they had only a year ago. 

Computer-based testing (or e-assessment) enables realization of trial test on 
convenience sample with simple administration and test analysis. It makes such 
studies much cheaper and easier to organize. Using Internet for computer-based 
testing, i.e. on-line testing, enables studies where communication with students 
and teachers who facilitate testing goes via Internet. 

Basic aims of this study were: 

1. to explore possibilities for on-line testing at national level in Serbia and 

2. to explore the trial test characteristics on a convenience sample in 
partially controlled conditions. 

This report presents evidences concerning possibilities for on-line testing as well 
as detailed analysis and comparison of results obtained from the annual and the 
trial test. 

Methods 

The on-line testing was realized in period May 5-19, 2009, immediately after the 
annual test in Nature & Society (April 2009). For the on-line testing modified 
Web application Moodle (with Ttest module) [2, 3] was used. Moodle had been 
used for all computer-based tests performed by the Institute in the past few 
years. 

The sample for the trial test was not specifically designed. All interested schools 
were enabled to participate. The only request we had was to choose participants 
among fourth-grade students who attend schools in Serbia. The schools were 
required to provide us with information about the way selection was done. 

Public media and direct e-mail communication were used to announce the on- 
line testing and to invite schools in Serbia to participate. Fifty six schools 
responded to our invitation (Appendix 2). Each school had a chance to select 
students for the test according to their own needs and opportunities. In total, the 
schools applied 1341 students. Accounts for the testing portal were made for all 
these students. Files with the Web address of testing portal, usernames, 
passwords, and the number of test intended for all the students were sent to all 
participating schools, as well as exact date and time when they should start 
testing. 

Two weeks before the trial testing, a short example-test was available for 
students at the testing portal so they could practice and become familiar with the 
testing procedure and the way of responding to the test. This example-test has 
been taken by 247 students and several curious teachers. 
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Finally, 926 students from 50 elementary schools participated in the trial test. 
Elementary school "Sveti Sava" from Batocina had the most participants (81 
students), while the least students had school "Djura Jaksic" from village Ravni 
(one student only). Because of numerous technical obstacles, like sudden 
interruption of Internet connection, obtained data is not valid for all the 
students. The trial test data analysis is performed for responses of 903 students. 

Conditions 

The on-line testing was deliberately performed in partially controlled conditions. 
Teachers in schools, where the testing was performed, had the opportunity to 
select students who would do the test. We gave instructions about the way 
teachers should realize testing, but we did not control the realization. 

The trial test was composed of four sections, with eight questions each. Using 
these sections, we have made four different versions of the same test with the 
same number of questions - 32. That was the way to help teachers to hinder 
students to peek into each other’s test. 

All the students were provided with test instructions. Short instructions had 
appeared on screen immediately after logging in the e-assessment portal. After 
the reading of short instructions, students listened to detailed instructions red by 
teacher. Students were not allowed to start their tests until teacher had finished 
the reading and finally gave them password to start the test. 

Computer classrooms in Serbian schools are equipped with computers with 
different keyboards and different language settings. Such a situation can cause 
problems with logging which may upset students at the very beginning of the 
testing [4]. Therefore, we decided to use numerical usernames and passwords. 
Once they start the test, there is no need for them to use keyboard any more. 
Students responded to all questions by using mouse, i.e. by selecting some 
alternative or option given on the screen. When students finished the test, they 
were able to see their total score. 

At the end, all students were asked to fulfill a short questionnaire (Appendix 4). 

Test 

All test questions were taken from questions pool used for the annual test in 
Nature & Society. The trial test had 32 questions (eight from each of four 
booklets at the annual test). Because of potential problems with a keyboard 
usage, only multiple choice and multiple response questions were chosen. 
Distribution of questions according to their difficulty and topics was similar to 
the distribution for the annual test. For administrative reasons, time limit for 
both tests was set to 50 minutes although students needed less time to answer 
all the questions. 

Feedback 

Computer-based testing, generally, enables automatic test scoring and a lot of 
feedback. Our trial-test had limited feedback so it would not influence 
performance and reliability of the study. After the students had submitted the 
answers, they were able to see their total scores but not which questions they 
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answered correctly. Teachers involved in testing realization had also a chance to 
see results of their students. For that purpose, all teachers had accounts with 
necessary privileges for approach to the system. 

Several days after the test was closed, all the questions and results became 
available to students and teachers. They used the same accounts for viewing the 
results as they used for taking the test earlier. 

Result analysis tools and methods 

Results of the trial and the annual tests were analyzed using Classical Test 
Theory (CTT) and Item Response Theory (IRT). Statistical analysis was done by 
using statistical software R with package irtoys [5] and standalone software IRT 
Command Language [6], Standard errors were determined by bootstrapping 
method. Estimation of students’ ability was done by using a posteriori method 
[7], 

We have separately analyzed all alternatives in multiple choice questions 
(including non-answers). In the case of multiple response questions, we have 
analyzed responses to all available options. For each alternative/option difficulty 
index, discrimination index and median response time were determined. 

Ttest module enables measuring responding time for each answer in the test. We 
have analyzed these data also in order to collect additional information about 
questions and the test as a whole. 

Two different scales were used for representing questions’ difficulty and 
students’ ability: the annual test scale and the trial test scale. Estimations of 
students’ ability are given on the annual test scale. For the analysis of 
alternatives we had used the trial test scale. 

Additional sources of information 

For this study we have used data collected for the annual test in Nature & Society 
and data collected in the preparation of the trial test for standard setting. Since 
the annual test did not have questionnaires for students and teachers like the 
trial test did, we used comparable data from students’ database maintained by 
Institute for Education Quality and Evaluation. 
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Comparison of results for the annual and the trial test 

Students who took the trial test, in average, had significantly higher achievement 
than students at the annual test. We can estimate ability of all students on both 
tests and present it on the same scale if we use IRT parameters obtained for the 
annual tests as benchmark. That way we can see how the convenience sample 
(used for the trial test) was different from the representative one (used for the 
annual test). 

The distributions of estimated ability for the annual and the trial test are given 
on Figure 1. We can see that both tests had samples with fairly normal 
distributions of similar width. These two distributions differ significantly by the 
mean value. 




I 1 1 1 1 1 1 

- 3 - 2-10123 



Figure 1: The distributions of estimated ability for 
the annual test (gray) and the trial test (green) 



Ability distribution for the annual test can be described as normal curve with the 
mean value p a = 0.00 and the standard deviation a a =0.88. For the trial test, the 
mean value is p t = 0.62 and the standard deviation o t =0.80. In other words, the 
same question’s difficulty parameter should be smaller at the trial test than at 
the annual test. That difference should be approximately the same as the 
difference between estimated abilities for these two tests. 

Comparison of questions' metric characteristics for the annual and 
the trial test 

If we do not have standard questions which we could use for the scale 
calibration, question parameters can be determined relatively for the sample 
that we have. That way, we can estimate parameters independently for the 
annual and the trial test. In two-parameter IRT model, in the ideal case, 
discrimination (parameter a) should have the same value regardless of the 
sample we use. Difficulty (parameter b ) for two tests should differ by constant 
value determined by the properties of used samples [8]. 
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The estimations of parameters a and b, as well as their standard deviations for 
the trial and the annual tests, are given in Table 1. We can notice that estimations 
of discrimination are approximately equal for both tests/samples, just as we 
expected. The estimation of questions’ difficulty, however, vary more than we 
expected. Differences are particularly big for very hard questions at the annual 
test (b a >2) and very easy questions at the trial test (h t <-2). This deviation from 
the theoretical expectation can be an evidence of possibility that these two tests 
are different not only because of different samples but because of other 
characteristics that we did not measure. 





Trial test 


Annual test 


Question 


a t 


bt 


«a 


b 3 


#1 


1. 1(0.1) 


-1. 1(0.1) 


1. 1(0.1) 


-0. 1(0.1) 


#2 


0.8(0. 1) 


0.8(0. 1) 


0.4(0. 1) 


3.4(0. 7) 


#3 


1. 7(0.2) 


-1.6(0. 1) 


1.4(0. 2) 


-1. 2(0.1) 


#4 


0.8(0. 1) 


1.8(0. 2) 


0. 7(0.1) 


1.9(0. 3) 


#5 


1. 0(0.1) 


-2.0(0. 2) 


1.0(0.1) 


-1.3(0. 2) 


#6 


0.8(0. 1) 


-0.9(0. 1) 


0.8(0. 1) 


-1.3(0. 2) 


#7 


0.8(0. 1) 


-0.6(0. 1) 


0. 5(0.1) 


1. 8(0.4) 


#8 


0.6(0. 1) 


-0.6(0. 2) 


0.6(0. 1) 


0.4(0. 2) 


#9 


0.6(0. 1) 


0.7(0. 2) 


0. 5(0.1) 


2.9(0. 6) 


#10 


0. 5(0.1) 


-0.8(0. 2) 


0. 5(0.1) 


0. 1(0.2) 


#11 


0. 7(0.1) 


0.4(0. 1) 


0.8(0. 1) 


1.8(0. 3) 


#12 


1.0(0. 1) 


-0.9(0. 1) 


0. 7(0.1) 


-0.7(0. 2) 


#13 


1.0(0. 1) 


-2.0(0. 2) 


1. 1(0.2) 


-1.6(0. 2) 


#14 


0. 7(0.1) 


-1.4(0. 2) 


0.8(0. 1) 


0.6(0. 2) 


#15 


0.9(0. 1) 


-1.5(0. 2) 


0. 7(0.1) 


0. 1(0.1) 


#16 


0. 7(0.1) 


1.7(0. 2) 


0.4(0. 1) 


3.7(0. 7) 


#17 


1. 0(0.1) 


-0. 5(0.1) 


1.0(0. 1) 


-0. 3(0.1) 


#18 


1.2(0. 1) 


-1. 1(0.1) 


1. 5(0.2) 


-0.4(0. 1) 


#19 


0. 5(0.1) 


-0.3(0. 2) 


0. 5(0.1) 


0.9(0. 3) 


#20 


0. 7(0.1) 


1.0(0. 2) 


0.6(0. 1) 


1.8(0. 3) 


#21 


0.9(0. 1) 


-l.O(O.l) 


0.8(0. 1) 


0.6(0. 2) 


#22 


0.8(0. 1) 


-1.7(0. 2) 


1. 1(0.2) 


-1.0(0. 2) 


#23 


1. 1(0.1) 


-1. 1(0.1) 


1. 5(0.2) 


-0.4(0. 1) 


#24 


0.9(0. 1) 


-0.6(0. 1) 


1.0(0. 1) 


0. 1(0.1) 


#25 


0.6(0. 1) 


1.8(0. 3) 


0. 7(0.1) 


2. 1(0.4) 


#26 


0.8(0. 1) 


-0. 5(0.1) 


0. 7(0.1) 


-0.3(0. 2) 


#27 


1. 0(0.2) 


-3. 1(0.5) 


0. 8(0.2) 


-3. 3(0. 6) 


#28 


0.9(0. 1) 


-1.9(0. 2) 


0.4(0. 1) 


-0.2(0. 3) 


#29 


0.9(0. 1) 


-2. 5(0. 3) 


1. 4(0.2) 


-1.8(0. 2) 


#30 


0. 7(0.1) 


-2. 5(0.4) 


0.8(0. 1) 


-1.2(0. 2) 


#31 


1. 3(0.2) 


-1. 3(0.1) 


1. 3(0.2) 


-0.6(0. 1) 


#32 


0.9(0. 1) 


-2. 7(0.4) 


1.0(0. 1) 


-1.6(0. 2) 



Table 1: The comparison of IRT parameters for all questions 
for the annual and the trial test 



The comparison of discrimination parameters for the two tests is given on Figure 
2. Dashed line displays theoretically expected values for the trial test, i.e. values 
equal to those at the annual test. The comparison of difficulty parameters for the 
two tests is given on Figure 3. Dashed line presents values from the annual test 
decreased by 0.62, i.e. expected value for the trial test. 
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Parameter a (annual test) 

Figure 2: The comparison of questions’ discrimination parameter (a) estimated for the 

annual and the trial test 




Parameter b (annual test) 

Figure 3: The comparison of questions’ difficulty parameter (b) estimated for the annual 

and the trial test 
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Number of omitted questions for the annual and the trial test 

There is a great difference between the number of omitted questions for annual 
and trial tests. In the first case, there are questions where up to 20% of students 
did not answer. In the second case, the percentage of omitted questions does not 
rise above 3%. This difference can be a consequence of the way of examination 
(mode of test delivery), but also of different way the selection of students for the 
testing was done. 

From the total number of non-answers for the trial test, 44 of them have 
response time 0 seconds. It means that students have not seen that question at 
all, so we can treat it as missed instead of omitted one. Since the response time 
can be measured for computer tests only, we can not measure the response time 
and know information about the number of missed questions for the annual test. 

The ratio of omitted questions for each item in both tests is presented in Table 2 
and on Figure 4. 



Question 


Question codename 


Ratio on 
non-answers 
fannual test! 


Ratio of 
non-answers 
ftrial test! 


#1 


PP031031 


0.03 


0.01 


#2 


PP013012 


0.02 


0.02 


#3 


PP041047 


0.01 


0.01 


#4 


PP061069 


0.01 


0.01 


#5 


PP071081 


0.02 


0.01 


#6 


PK093104 


0.02 


0.03 


#7 


PK123146 


0.17 


0.01 


#8 


PD171192 


0.05 


0.01 


#9 


PP033036 


0.01 


0.01 


#10 


PP013010 


0.01 


0.01 


#11 


PP043051 


0.03 


0.01 


#12 


PP063072 


0.03 


0.01 


#13 


PP073082 


0.02 


0.01 


#14 


PK111131 


0.13 


0.01 


#15 


PK121136 


0.13 


0.01 


#16 


PK123144 


0.15 


0.02 


#17 


PP043052 


0.03 


0.02 


#18 


PP061067 


0.02 


0.01 


#19 


PK093107 


0.02 


0.01 


#20 


PK101113 


0.02 


0.01 


#21 


PK123142 


0.20 


0.02 


#22 


PD141159 


0.02 


0.01 


#23 


PD133154 


0.11 


0.01 


#24 


PP013009 


0.03 


0.02 


#25 


PP083095 


0.05 


0.02 


#26 


PK101118 


0.02 


0.02 


#27 


PK111132 


0.01 


0.01 


#28 


PK121139 


0.14 


0.01 


#29 


PD143165 


0.02 


0.01 


#30 


PD153176 


0.03 


0.02 


#31 


PP021015 


0.02 


0.01 


#32 


PK093106 


0.01 


0.01 



2 i r i 

3 □ 

4 □ 

5 i n 

6 I 

7 L H 

8 |l I 

9 mr 

10 rr 

11 i r ~i 

12 !,i i 

13 I I 

14 I I 



18 ! _ LJ 

19 I □ 

20 II 1 I 

21 I 

22 | _l I 

23 I ~ I 

24 

25 r__L.. 

26 I 

27 I I 

28 I I 

29 I □_ 

30 I F~ 

31 I I 

32 |H| 



0.00 0.05 0.10 0.15 0.20 



Figure 4: The ratio of non-answers for the annual (gray) 
and the trial test (green) 
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DIF by gender 

Differential Item Functioning (DIF) for examinees grouped by gender is analyzed 
using Mantel-Haenszel method [9]. DIF p-value is determined separately for the 
annual and the trial test. All items where DIF p-value is less than 0.05 behaves 
statistically different for boys and girls. This discrepancy does not need to be an 
indicator of item bias, but it is a reason to check what and how that item 
measures. 

p-Values for the annual and the trial test are presented in Table 3 and on Figure 
5. Here we can see that items #18 and #23 are DIF flagged at both representative 
and convenience sample, while items #10, #17, and #25 are not. For item #23 
significance of DIF difference is biggest for both tests (p<0,01). 



Question 


Question codename 


DIF p-value 
(annual test) 


DIF p-value 
(trial test) 


#1 


PP031031 


0.98 


0.95 


#2 


PP013012 


0.92 


0.95 


#3 


PP041047 


0.60 


0.77 


#4 


PP061069 


0.39 


0.92 


#5 


PP071081 


0.07 


0.38 


#6 


PK093104 


0.02 


0.23 


#7 


PK123146 


0.53 


0.62 


#8 


PD171192 


0.53 


0.20 


#9 


PP033036 


0.15 


0.93 


#10 


PP013010 


0.98 


0.04 


#11 


PP043051 


0.37 


0.77 


#12 


PP063072 


0.24 


0.80 


#13 


PP073082 


0.38 


0.87 


#14 


PK111131 


0.26 


0.43 


#15 


PK121136 


0.11 


0.31 


#16 


PK123144 


0.97 


0.51 


#17 


PP043052 


0.31 


0.04 


#18 


PP061067 


0.04 


0.05 


#19 


PK093107 


0.95 


0.36 


#20 


PK101113 


0.27 


0.28 


#21 


PK123142 


0.70 


0.84 


#22 


PD141159 


0.27 


0.88 


#23 


PD133154 


0.00 


0.00 


#24 


PP013009 


0.92 


0.13 


#25 


PP083095 


0.46 


0.05 


#26 


PK101118 


0.51 


0.55 


#27 


PK111132 


0.70 


0.56 


#28 


PK121139 


0.39 


0.97 


#29 


PD143165 


- 


0.92 


#30 


PD153176 


0.77 


0.89 


#31 


PP021015 


0.44 


0.97 


#32 


PK093106 


0.81 


0.61 




0.002 0.005 0.010 0.020 0.050 0.100 0.200 0.500 1.000 

DIF p-value (annual test) 

Figure 5: DIF p-values for the annual and the trial test 
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Conclusions 



On-line test in Nature & Society is realized using computer-based testing system 
where students participated from their own schools through the Internet. All 
elementary schools in Serbia had opportunity to participate in this testing if the 
school had an interest and technical capabilities. The organization of on-line 
testing, as an innovation in educational practice, was accompanied by many 
technical problems. Because of the quality and reliability of Internet connections, 
such a testing was impossible only two years ago. Today, we can say that on-line 
testing is feasible in the majority of schools in Serbia. The quality of Internet 
connections in schools is still the biggest obstacle in the realization of e-testing. 
Due to technical difficulties, in 6 out of 56 engaged schools, testing was not 
finished successfully. The second biggest obstacle for e-testing was the quality of 
Institute’s Internet connection where the e-testing server is located. 

We have received the feedback concerning testing realization from 25 teachers. 
14 Teachers said there were no technical problems of any kind, 3 of them said 
that some pages were loading slowly, 5 of them said that Internet connection 
was interrupted but it did not violate the testing, while 3 teachers said they 
encountered too many technical problems so they can not say the testing was 
successful. 22 Teachers said that students were very motivated to answer all the 
questions as good as they could. 19 Teachers said that praxis of e-testing in 
Serbia would be very useful for all students while the others thought that just 
certain students would benefit from such a praxis. Generally, teachers were very 
interested in this kind of testing and they hoped such testing would be available 
more often in the future. 

Psychometric characteristics of the trial test were examined through the results 
of both the annual and the trial test. The characteristics of individual items in the 
trial test were compared with the same item in the annual test. Results of both 
tests were analyzed using Classical Testing Theory (CTT) and 2-parameter Item 
Response Theory (2PL IRT). The annual test was performed on representative 
sample (1842 students in total) in paper & pencil mode, while the sample for the 
trial test was convenience sample (926 students from 50 elementary schools). 
Because of unavoidable technical difficulties in the e-testing realization, some 
test taking attempts were not valid. Finally, 903 students had valid responses 
suitable for the analysis. 

Trial test was composed of 32 questions (eight from each of four annual test 
booklets). All selected items were of multiple-choice type (with one or more 
correct answers). The distribution of items by topics and difficulties were 
approximately the same as for the annual test. Responding time for both test was 
limited to 50 minutes. 

According to Item Response Theory, we can reconstruct item parameters for 
entire population if we know parameters for a sample that is large enough and 
normally distributed, and standard items for scale calibration. That means that 
estimation of item discrimination, in ideal case, should have the same value for 
all normally distributed samples, while item difficulty estimation should be 
linearly shifted in dependence on the mean ability of used sample. In this study, 
we have examined how good can we estimate item parameters for the entire 
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population on the basis of on-line trial test given to convenience sample of 
students. The assumed similarity of item response curves for these two tests 
opens another research question - Can we detect differential item functioning 
for entire population if we look at the results of the trial test only? 

Results show that differences in the estimation of item discrimination for the 
annual and the trial test are small and comparable to their standard errors. For 
item difficulty we can see that there is linear dependence between estimations 
obtained for these two tests with correlation coefficient r=0.89. Such a 
correlation with theoretical model, however, does not enable us to make good 
prediction of parameters for individual items. Difference in testing conditions 
certainly have great influence on the estimation of individual item parameters. 
(We have some indications that conditions in certain schools were not 
completely regular. That is also a reason why results do not fit in theoretical 
model as good as we expected.) Results of numerical simulations show that these 
two tests differ more than can be explained by the difference in samples. 
Obviously, there is a difference in students’ attitude toward the tests. This 
discrepancy is reflected in the number of non-answered questions for the annual 
(5.0% per student in average) and the trial test (1.3%). The analysis of 
differential item functioning shows that two out of three items are flagged 
correctly (DIF p-value less than 0.05) for the annual test using the results of the 
trial test only. 

In this study we have also analyzed alternatives, i.e. options for multiple choice 
questions. The analysis of alternatives frequency against estimated ability can 
help us detect bad alternatives or strong distractors. Information about item 
response time fulfills this picture because it enables us to detect strong 
distractors even when difficulty and discrimination parameters of alternatives 
do not indicate such behavior (e.g. the fourth alternative of question #9). 

The general conclusion of this study is that on-line testing is feasible, that 
teachers need them and that such a practice motivate students. In psychometric 
domain, on-line trial tests in partially controlled conditions are useful for item 
and test quality analysis and usability of alternatives in multiple choice 
questions. The estimation of item difficulty is also possible but under constrains 
that have to be thoroughly examined in the future. 
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Appendix 1: Metric characteristics of items 

All items from the trial test and their metric characteristics are given in this 
Appendix. Number and codename of items are displayed in the header of each 
page. Response codes ©, ©, ©, ® are associated with alternatives in multiple- 
choice questions. Code (D is reserved for non-answers. Correct (V) and incorrect 
answers (*) are labeled for all alternatives. If there is a question with a few 
correct answers, we consider it successfully answered if examinee responded 
correctly to all options like it was set for the annual test. 

CTT parameters are determined for all items (item difficulty - p-value and item 
discrimination - r-value). Item difficulty is determined as the ratio of number of 
correct answers and the total number of answers for particular item. Item 
discrimination is obtained as item-total correlation with exclusion. 2PL IRT 
parameters are also determined for all items (difficulty - b and discrimination - 
a). For the sake of comparison, all item parameters are determined for both: the 
annual and the trial test. 

For the trial test, all alternatives (or options for multiple response questions) 
were analyzed separately. The number of responses, r-value, p-value and 
characteristic response time are given in tables for all alternatives or options. 
Characteristic response time is given in seconds as the median value of response 
time distribution for each alternative or option. 

Frequency curves for particular alternatives, or options, are given graphically for 
the trial test. Here we used non-calibrated scale of ability determined for the trial 
test. The curves of alternatives or options are colored according to the response 
code. 
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Question #1 



PP031031 



y npwpoflHa craHHLUTa y6pajaMo: 
Ofla6epw jeflaH oflroBop. 



o 


peKy, i-bMBy m je3epo 


X 


© 


o 


myMy, peKy m 6apy 




© 


o 


BMHorpafl, cjDapMy v\ BofiibaK 


X 


© 


o 


riMBafly, tbHBy m noBpTtbaK 


X 






(no response) 


X 


© 



Item parameters 



CTT 




IR r 


r 




r-value 


p-value 






a 


b 


annual test 


0.40 


0.51 




annual test 


1.1 


-0.1 


trial test 


0.35 


0.72 




trial test 


1.1 


-1.0 



Analysis of alternatives 



Response 

code 


Number of 
responses 


p-value of 
alternative 


r-value of 
alternative 


Median response 
time [s] 


© 


65 


0.07 


-0.14 


43 


© 


639 


0.72 


0.35 


39 


© 


80 


0.09 


-0.18 


41 


© 


94 


0.11 


-0.18 


40 




10 


0.01 


-0.10 


76 



PP031031 




Ability 



14 




Question #2 



PP013012 



Kopetbe 6n/baKa cnpeiaBa: 
Ofla6epn jeflaH oflroBop. 



o 


cnupatbe 3eM/bmiJTa 




© 


o 


3ara1)MBatbe 3eM/bmiJTa 


X 


© 


o 


ncywMBatbe 3eM/bmiJTa 


X 


© 


o 


T)y6petbe 3eM/bHLiJTa 


X 


@ 




(no response) 


X 


© 



Item parameters 



CTT 




IR1 






r-value 


p-value 






a 


b 


annual test 


0.09 


0.23 




annual test 


0.4 


3.4 


trial test 


0.28 


0.37 




trial test 


0.8 


0.8 



Analysis of alternatives 



Response 

code 


Number of 
responses 


p-value of 
alternative 


r-value of 
alternative 


Median response 
time fs] 


© 


327 


0.37 


0.28 


46 


© 


230 


0.26 


-0.13 


43 


© 


221 


0.25 


-0.05 


48 


© 


93 


0.10 


-0.15 


39 




17 


0.02 


-0.08 


58 



PP013012 




Ability 



15 




Question #3 



PP041047 



UlTa Ofl HaBefleHor He y6pajaMO y npupoflHa 6oraTCTBa? 
Ofla6epw jeflaH oflroBop. 



o 


6mbHH M >KMBOTW-bCKM CBeT 


X 


© 


o 


Kyfie 




© 


o 


pyfle 


X 


© 


o 


BOfly 


X 






(no response) 


X 


© 



Item parameters 



CTT 




IR r 






r-value 


p-value 






a 


b 


annual test 


0.44 


0.77 




annual test 


1.4 


-1.2 


trial test 


0.39 


0.87 




trial test 


1.7 


-1.6 



Analysis of alternatives 



Response 

code 


Number of 
responses 


p-value of 
alternative 


r-value of 
alternative 


Median response 
time [s] 


© 


36 


0.041 


-0.19 


40 


© 


770 


0.87 


0.39 


27 


© 


51 


0.06 


-0.26 


34 


© 


23 


0.03 


-0.14 


43 




8 


0.01 


-0.14 


37 



PP041047 




Ability 



16 




Question #4 



PP061069 



Koja CBOjcTBa MMa Ba3flyx? 
Ofla6epn 6ap je,qaH oflroBop. 



□ 


MM3 CTa/iaH o6jimk 


X 


O 


□ 


Kpelie ce 




© 


□ 


3ay3MMa npocrop 




© 


□ 


MM3 6ojy 


X 


© 


□ 


y racoBMTOM je crabby 




© 



Item parameters 



CTT 




IR1 






r-value 


p-value 






a 


b 


annual test 


0.25 


0.23 




annual test 


0.7 


1.9 


trial test 


0.27 


0.22 




trial test 


0.8 


1.8 



Analysis of options 



Option 


Number of 
responses 


p-value of option 


r-value of option 


Median response 
time [s] 


O 


191 


0.22 


-0.14 


41 


© 


555 


0.63 


0.10 


44 


© 


403 


0.46 


0.25 


39 


© 


17 


0.02 


-0.11 


35 




657 


0.74 


0.32 


41 



PP061069 




Ability 



17 




Question #5 



PP071081 



IllTa lie ce flecmn aKO y nocyfly ca boaom craBMMO flBe nonTMue: jeflHy ofl 

cmponopa, a flpyry ofl craKna? 

Ofla6epw jeflaH oflroBop. 



o 


06e fie noTOHyTM. 


X 


© 


o 


06e fie nnyTaTM. 


X 


© 


o 


Jlorrmua ofl CTMponopa fie noTOHyTM, a craicneHa fie nnyTaTM. 


X 


© 


o 


JlonTMua ofl CTMponopa fie nnyTaTM, a craKneHa fie noTOHyTM. 








(no response) 


X 


© 



Item parameters 



CTT 




IR r 






r-value 


p-value 






a 


b 


annual test 


0.36 


0.75 




annual test 


1.0 


-1.3 


trial test 


0.28 


0.83 




trial test 


1.0 


-2.0 



Analysis of alternatives 



Response 

code 


Number of 
responses 


p-value of 
alternative 


r-value of 
alternative 


Median response 
time [s] 


© 


21 


0.02 


-0.15 


42 


© 


54 


0.06 


-0.17 


51 


© 


65 


0.07 


-0.12 


50 


© 


736 


0.83 


0.27 


42 




12 


0.01 


-0.07 


60 



PP071081 




- 2-10 1 2 
Ability 



18 




Question #6 



PK093104 





Ha Kojoj cjimum je TaHHO npm<a3aH nonowaj 
cem<e? 

Ofla6epw jeflaH oflroBop. 





o 


a 


X 


© 


--0- 


o 


6 


X 


© 




o 


B 


V 


© 


GT 


o 


r 


X 




r 


(no response) 


X 


© 



Item parameters 



CTT 




IR r 






r-value 


p-value 






a 


b 


annual test 


0.31 


0.71 




annual test 


0.8 


-1.3 


trial test 


0.30 


0.66 




trial test 


0.8 


-0.9 



Analysis of alternatives 



Response 

code 


Number of 
responses 


p-value of 
alternative 


r-value of 
alternative 


Median response 
time [s] 


© 


141 


0.16 


-0.24 


53 


© 


71 


0.08 


-0.11 


51 


© 


584 


0.66 


0.31 


43 


© 


67 


0.08 


-0.01 


54 




25 


0.03 


-0.15 


50 



PK093104 




- 2-10 1 2 



Ability 



19 




Question #7 



PK123146 



IllTa je 6 mo hoboa 3a M36njahbe npBor cpncKor yciaHKa? 
Ofla6epw jeflaH oflroBop. 



o 


flaHaK y kpbh 


X 


© 


o 


ceia KHe30Ba 




© 


o 


3Mflatbe Tie/ie Kyne 


X 


© 


o 


ceo6a Cp6a 


X 






(no response) 


X 


© 



Item parameters 



CTT 




IR r 






r-value 


p-value 






a 


b 


annual test 


0.18 


0.31 




annual test 


0.5 


1.8 


trial test 


0.31 


0.60 




trial test 


0.8 


-0.6 



Analysis of alternatives 



Response 

code 


Number of 
responses 


p-value of 
alternative 


r-value of 
alternative 


Median response 
time fs] 


© 


140 


0.16 


-0.07 


38 


© 


529 


0.60 


0.31 


30 


© 


40 


0.05 


-0.10 


49 


© 


167 


0.19 


-0.23 


34 




12 


0.01 


-0.16 


42 



PK123146 




Ability 



20 




Question #8 



PD171192 




Ko ce Kpefie npaBM/mo ynnuoM 6e3 TpoToapa? 
Ofla6epw jeflaH ouroBop 



o 


neinaK A S 


© 


o 


nemaK B x 


© 


o 


h neinaK A n neinaK 5 x 


© 


o 


hh neinaK A hh neinaK B x 






(no response) x 


© 



Item parameters 



CTT 




IR r 






r-value 


p-value 






a 


b 


annual test 


0.23 


0.45 




annual test 


0.6 


0.4 


trial test 


0.23 


0.58 




trial test 


0.6 


-0.6 



Analysis of alternatives 



Response 

code 


Number of 
responses 


p-value of 
alternative 


r-value of 
alternative 


Median response 
time [si 


© 


519 


0.58 


0.23 


44 


© 


243 


0.27 


-0.09 


42 


© 


27 


0.03 


-0.18 


48 


© 


90 


0.10 


-0.11 


46 




9 


0.01 


-0.09 


75 



PD171192 




Ability 



21 




Question #9 



PP033036 



Kojn ce ycnoBM 3a >kmbot y CTaHmuTMMa HajBmue Met-bajy tokom roamne? 



Ofla6epn jeflaH 



OflroBop 








O 


CBeT/iocT h TonnoTa 




© 


O 


KO/lMHMHa BOfle 


X 


© 


o 


KO/iMHHHa Ba3flyxa 


X 


© 


o 


6poj >kmbmx 6wTia 


X 






(no response) 


X 


© 



Item parameters 



CTT 




IR1 






r-value 


p-value 






a 


b 


annual test 


0.14 


0.23 




annual test 


0.5 


2.9 


trial test 


0.21 


0.42 




trial test 


0.6 


0.7 



Analysis of alternatives 



Response 

code 


Number of 
responses 


p-value of 
alternative 


r-value of 
alternative 


Median response 
time [s] 


0 


370 


0.42 


0.22 


50 


© 


68 


0.08 


0.00 


55 


© 


35 


0.04 


-0.12 


52 


© 


404 


0.45 


-0.16 


40 




11 


0.01 


-0.05 


26 



PP033036 




Ability 



22 




Question #10 



PP013010 



Ha TpaBM je craja/ia KaHTa ca T)y6peT0M. Ilocne HeKO/iMKO flaHa TpaBa ncnofl 
KaHTe je no>KyTe/ia. 36or Hera? 

Ofla6epn jeflaH oflroBop. 



o 


Htije MMa/ia flOBO/bHO BOfle. 


X 


© 


o 


Hwje HMa/ia flOBO/bHO Ba3flyxa. 


X 


© 


o 


Hwje MMa/ia flOBO/bHO cyi-weBe CBeTJiocm. 


V 


© 


o 


HHje MMana flOBO/bHO TonnoTe. 


X 


@ 




(no response) 


X 


© 



Item parameters 



CTT 




IR r 






r-value 


p-value 






a 


b 


annual test 


0.20 


0.49 




annual test 


0.5 


0.1 


trial test 


0.21 


0.60 




trial test 


0.5 


-0.8 



Analysis of alternatives 



Response 

code 


Number of 
responses 


p-value of 
alternative 


r-value of 
alternative 


Median response 
time [s] 


© 


111 


0.12 


-0.16 


46 


© 


212 


0.24 


-0.04 


46 


© 


528 


0.59 


0.20 


42 


© 


25 


0.03 


-0.13 


43 


<D 


12 


0.01 


-0.07 


44 



PP013010 




- 2-10 1 2 



Ability 



23 




Question #11 



PP043051 



Koje npnpoflHO 6oraTCTBO jbyflM KopwcTe m Kao M3Bop eHepmje v\ Kao 
CkipOBHHy? 

Ofla6epw jeflaH oflroBop. 



rac 


X 


© 


flpeo 


V 


© 


KaMeH 


X 


© 


3eMJbHWTe 


X 


@ 


(no response) 


X 


© 



Item parameters 



CTT 




IR1 






r-value 


p-value 






a 


b 


annual test 


0.24 


0.23 




annual test 


0.8 


1.8 


trial test 


0.26 


0.45 




trial test 


0.7 


0.4 



Analysis of alternatives 



Response 

code 


Number of 
responses 


p-value of 
alternative 


r-value of 
alternative 


Median response 
time [si 


© 


343 


0.39 


-0.07 


39 


© 


400 


0.45 


0.26 


40 


© 


16 


0.02 


-0.05 


45 


© 


117 


0.13 


-0.25 


38 




12 


0.01 


-0.04 


38 



PP043051 




Ability 



24 




Question #12 



PP063072 



JeflHor neTht>er jyTpa MapKO je npMMeTMO fla cy cbm n/ioiHMUM cyBM, a aa Ha 
TpaBM y 6aiuTaMa mm3 Kan/bMua BOfle. HMMe ce to MO>Ke o6jacHMTM? 

Ofla6epM jeflaH oflroBop. 

O nafla/ia je KMiua. x © 

O naflao je rpafl. x © 

O TpaBa je nycm/ia coKOBe. x © 

O Ha TpaBM ce 3rycHyjia BOfleHa napa. S © 

(no response) * ® 



Item parameters 



CTT 




IR r 






r-value 


p-value 






a 


b 


annual test 


0.28 


0.60 




annual test 


0.7 


-0.7 


trial test 


0.35 


0.69 




trial test 


1.0 


-0.9 



Analysis of alternatives 



Response 

code 


Number of 
responses 


p-value of 
alternative 


r-value of 
alternative 


Median response 
time [s] 


© 


155 


0.17 


-0.29 


54 


© 


29 


0.03 


-0.12 


56 


© 


84 


0.09 


-0.07 


58 


© 


607 


0.68 


0.36 


52 




13 


0.01 


-0.11 


66 



PP063072 




- 2-10 1 2 
Ability 



25 




Question #13 



PP073082 



3aiuT0 ce nocyne 3a KyBatbe xpaHe npaBe o,q MeTa/ia? 



Ofla6epn jeflaH oflroBop. 

3aTO luto MeTa/i flo6po npoBO.au Ton/icrry S © 

3aTO luto MeTa/i npoBOflu e/ieKTpnMHy CTpyjy x © 

O 3aTO llito ce MeTa/i He jiomm x ® 

3aTO luto MeTa/i Hkije npoBMflaH x © 

(no response) x ® 



Item parameters 



CTT 




IR r 






r-value 


p-value 






a 


b 


annual test 


0.39 


0.81 




annual test 


1.1 


-1.6 


trial test 


0.30 


0.83 




trial test 


1.1 


-2.0 



Analysis of alternatives 



Response 

code 


Number of 
responses 


p-value of 
alternative 


r-value of 
alternative 


Median response 
time [s] 


© 


739 


0.83 


0.30 


40 


© 


43 


0.05 


-0.12 


39 


© 


83 


0.09 


-0.24 


44 


© 


12 


0.01 


-0.06 


45 




11 


0.01 


-0.10 


47 



PP073082 




Ability 



26 




Question #14 



PK111131 



KocoBCKa 6nTKa ce oflnrpa/ia 1389 . roflMHe. Kojn je to BeK? 
Ofla6epw jeflaH oflroBop. 



o 


jeflaHaecTM BeK 


X 


© 


o 


ABaHaecTM BeK 


X 


© 


o 


TpMHaeCTM BeK 


X 


© 


o 


neTpHaecTM BeK 


V 






(no response) 


X 


© 



Item parameters 



CTT 




IR r 






r-value 


p-value 






a 


b 


annual test 


0.29 


0.40 




annual test 


0.8 


0.6 


trial test 


0.27 


0.71 




trial test 


0.7 


-1.4 



Analysis of alternatives 



Response 

code 


Number of 
responses 


p-value of 
alternative 


r-value of 
alternative 


Median response 
time [s] 


© 


26 


0.03 


-0.19 


45 


© 


83 


0.09 


-0.11 


34 


© 


137 


0.15 


-0.13 


40 


© 


631 


0.71 


0.27 


31 




11 


0.01 


-0.10 


38 



PK111131 




Ability 



27 




Question #15 



PK121136 



Ko je 6mo Bo1)a npBor cpncKor ycTaHKa? 
Ofla6epw jeflaH oflroBop. 



o 


Mmjiouj 06peHOBM?i 


X 


© 


o 


Kapalppfje 




© 


o 


Uap flymaH 


X 


© 


o 


neTap ripBM KapaTjopfjeBMfi 


X 






(no response) 


X 


© 



Item parameters 



CTT 




IR r 






r-value 


p-value 






a 


b 


annual test 


0.28 


0.49 




annual test 


0.7 


0.1 


trial test 


0.29 


0.75 




trial test 


0.9 


-1.5 



Analysis of alternatives 



Response 

code 


Number of 
responses 


p-value of 
alternative 


r-value of 
alternative 


Median response 
time fs] 


© 


74 


0.08 


-0.15 


27 


© 


667 


0.75 


0.29 


24 


© 


50 


0.06 


-0.21 


30 


© 


86 


0.10 


-0.08 


30 




11 


0.01 


-0.11 


37 



PK121136 




Ability 



28 




Question #16 



PK123144 



Koje cy TBpflhbe o >kmboty y BpeMe HeMai-bulia TaHHe? 
Ofla6epw 6ap jeflaH oflroBop. 



□ 


HajBeTiH 6poj craHOBHMKa hmhmjim cy ce/bauti 




O 


□ 


KyTie cy o6i/nho 6wie rpafjeHe o,q unra/ia 


X 


© 


□ 


CBa flepa cy nw/ia y inKony 


X 


© 


□ 


HajBefm 6poj /byflu 6aBno ce TproBMHOM 


X 


© 


□ 


nocylje ce npaBwio oa flpBeia m neiei-ie r/iMHe 




© 



Item parameters 



CTT 




IR1 






r-value 


p-value 






a 


b 


annual test 


0.10 


0.20 




annual test 


0.4 


3.7 


trial test 


0.25 


0.27 




trial test 


0.7 


1.7 



Analysis of options 



Option 


Number of 
responses 


p-value of option 


r-value of option 


Median response 
time fs] 


O 


600 


0.68 


0.31 


65 


© 


134 


0.15 


-0.19 


72 


© 


133 


0.15 


-0.22 


67 


© 


309 


0.35 


0.10 


68 




576 


0.65 


0.33 


65 



PK123144 




Ability 



29 




Question #17 



PP043052 



3aiuT0 je BOfla o6HOB/bHB M3Bop eHepmje? 
Ofla6epw jeflaH oflroBop. 



o 


3aTO LLiTO ce Ha/ia3M CByna oko Hac. 


X 


© 


o 


3aTO LLiTO ce Mowe Hafm y Tpn arperaTHa cratba. 


X 


© 


o 


3aTO urn) je mm 3 HajBmije Ha 3eM/bn. 


X 


© 


o 


3aTO LLiTO HenpeKMflHO Kpy>KM y npnpo,an. 







(no response) * © 



Item parameters 



CTT 




IR r 






r-value 


p-value 






a 


b 


annual test 


0.37 


0.56 




annual test 


1.0 


-0.3 


trial test 


0.33 


0.61 




trial test 


1.0 


-0.5 



Analysis of alternatives 



Response 

code 


Number of 
responses 


p-value of 
alternative 


r-value of 
alternative 


Median response 
time fs] 


© 


134 


0.15 


-0.26 


42 


© 


95 


0.11 


-0.01 


47 


© 


104 


0.12 


-0.15 


41 


© 


541 


0.61 


0.33 


38 




14 


0.02 


-0.14 


39 



PP043052 




Ability 



30 




Question #18 



PP061067 



Koje je 3ajeflHMHKO CBojcTBO BOfle m cbmx flpymx tghhoctm? 
Ofla6epw jeflaH oflroBop. 



o 


HeMa 6ojy 


X 


© 


o 


HeMa CTa/iaH o6jimk 




© 


o 


HeMa MMpMC 


X 


© 


o 


HeMa yxyc 


X 






(no response) 


X 


© 



Item parameters 



CTT 




IR r 






r-value 


p-value 






a 


b 


annual test 


0.50 


0.60 




annual test 


1.5 


-0.4 


trial test 


0.36 


0.73 




trial test 


1.2 


-1.1 



Analysis of alternatives 



Response 

code 


Number of 
responses 


p-value of 
alternative 


r-value of 
alternative 


Median response 
time [s] 


© 


54 


0.06 


-0.18 


44 


© 


649 


0.73 


0.35 


37 


© 


106 


0.12 


-0.18 


43 


© 


67 


0.08 


-0.19 


38 




12 


0.01 


-0.04 


70 



PP061067 




Ability 



31 




Question #19 



PK093107 



36or nera fie 3ry>KBaHn jimct narmpa 6p>Ke nacm Ha t/io Hero hcth tmct 
narmpa Kojn HHje 3ry>KBaH? 

Ofla6epn jeflaH oflroBop. 



o 


36or 06/iMKa 




© 


o 


36or xpanaBOcm 


X 


© 


o 


36or fle6/bi/me 


X 


© 


o 


36or Te>KMHe 


X 






(no response) 


X 


© 



Item parameters 



CTT 




IR r 






r-value 


p-value 






a 


b 


annual test 


0.17 


0.41 




annual test 


0.5 


0.9 


trial test 


0.20 


0.54 




trial test 


0.5 


-0.3 



Analysis of alternatives 



Response 

code 


Number of 
responses 


p-value of 
alternative 


r-value of 
alternative 


Median response 
time [s] 


© 


474 


0.53 


0.19 


42 


© 


15 


0.02 


-0.09 


37 


© 


26 


0.03 


-0.02 


46 


© 


361 


0.41 


-0.15 


40 


<D 


12 


0.01 


-0.05 


22 



PK093107 
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Question #20 



PK101113 




Ako je Kyfia Ha MCTOKy, Ha Kojoj 
CTpaHM CBeTa je myMa y OflHOcy Ha 
KyTiy? 

Ofla6epn jeflaH oflroBop. 



o 


Ha jyry 


s 


© 


o 


Ha ceBepy 


X 


© 


o 


Ha 3anafly 


X 


© 


o 


Ha HCTOKy 


X 




(no 


response) 


X 


© 



Item parameters 



CTT 




IR1 






r-value 


p-value 






a 


b 


annual test 


0.20 


0.28 




annual test 


0.6 


1.8 


trial test 


0.26 


0.35 




trial test 


0.7 


1.0 



Analysis of alternatives 



Response 

code 


Number of 
responses 


p-value of 
alternative 


r-value of 
alternative 


Median response 
time [s] 


© 


312 


0.35 


0.26 


47 


© 


192 


0.22 


0.00 


50 


© 


304 


0.34 


-0.14 


41 


© 


67 


0.08 


-0.20 


46 




13 


0.01 


-0.05 


54 



PK101113 
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Question #21 



PK123142 



Uap flymaH je 6mo Be/iMKM ocBajan m moTihm B/ia,qap Cp6nje. 36or Tora je y 
Hapofly 3anaMfieH Kao AywaH cm/ihm. 36or Hera je hberoBor cm-ia Hapofl 
Ha3Bao ypoui HejaKM? 

Ofla6epn je/iaH o,qroBop. 



o 


Bmo je HajM/iaf)M flywaHOB cmh. 


X 


© 


o 


Bmo je He>KHe m KpxKe rpafje. 


X 


© 


o 


3a BpeMe hberoBe Bna,qaBMHe HMje flOHeT hobm 3aKOHMK. 


X 


© 


o 


3a BpeMe hberoBe Bna,qaBMHe Beni/iKaniM cy pacnapia/iM 


Y 






uapcTBO. 








(no response) 


X 


© 



Item parameters 



CTT 




IR r 






r-value 


p-value 






a 


b 


annual test 


0.30 


0.40 




annual test 


0.8 


0.6 


trial test 


0.32 


0.69 




trial test 


0.9 


-1.0 



Analysis of alternatives 



Response 

code 


Number of 
responses 


p-value of 
alternative 


r-value of 
alternative 


Median response 
time [s] 


© 


94 


0.11 


-0.28 


56 


© 


102 


0.11 


-0.07 


56 


© 


68 


0.08 


-0.13 


56 


© 


611 


0.69 


0.32 


48 




13 


0.01 


-0.07 


61 



PK123142 
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Question #22 



PD141159 



UlTa cy ofl HaBefleHor cmm6oum CBaKe flpwaBe? 
Ofla6epw 6ap jeflaH oflroBop. 



□ 


TepmopMja 


X 


O 


□ 


XMMHa 




© 


□ 


rnaBHM rpafl 


X 


© 


□ 


3acTaBa 




© 


□ 


rp6 




e 



Item parameters 



CTT 




IR r 






r-value 


p-value 






a 


b 


annual test 


0.39 


0.71 




annual test 


1.1 


-1.0 


trial test 


0.26 


0.77 




trial test 


0.8 


-1.7 



Analysis of options 



Option 


Number of 
responses 


p-value of option 


r-value of option 


Median response 
time fs] 


O 


53 


0.06 


-0.08 


47 


© 


791 


0.89 


0.31 


37 


© 


71 


0.08 


-0.12 


46 


© 


807 


0.91 


0.32 


38 




807 


0.91 


0.31 


38 



PD141159 




Ability 
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Question #23 



PD133154 



AHa m Mn/inpa cy ce nocBafja/ie 3a BpeMe LUKO/iCKor oflMopa. AHa je Tyx<n.na 
Mn/inpy KOfl ynnTe/bnue. YHHTe/bHua je n3rp,qnjia Mnnnuy, He cac/iyLuaBLun LUTa 
OHa mm3 fla Ka>Ke o TOMe. Koje ce flenje npaBO He nouiTyje y obom npnMepy? 

Ofla6epn je/iaH onroBop. 



o 


npaBO Ha cno6oflHO BpeMe h nrpy 


X 


© 


o 


npaBO Ha 3amTHTy Kafla nx oflpacni/i 3/iocTaB/bajy 


X 


© 


o 


npaBO Ha n3pa>xaBatbe B/iacTMTor MHw/betba 


V 


© 


o 


npaBO Ha rmiHe TajHe 


X 


@ 




(no response) 


X 


© 



Item parameters 



CTT 




IR r 






r-value 


p-value 






a 


b 


annual test 


0.51 


0.61 




annual test 


1.5 


-0.4 


trial test 


0.36 


0.72 




trial test 


1.1 


-1.1 



Analysis of alternatives 



Response 

code 


Number of 
responses 


p-value of 
alternative 


r-value of 
alternative 


Median response 
time [s] 


© 


73 


0.08 


-0.19 


64 


© 


103 


0.12 


-0.18 


62 


© 


639 


0.72 


0.36 


57 


© 


61 


0.07 


-0.17 


51 




12 


0.01 


-0.06 


61 



PD133154 
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Question #24 



PP013009 



>KnBOTMhbe HacTanbeHe y 3eM/bmuTy noMa>Ky fla oho 6yfle: 
Ofla6epn jeflaH oflroBop. 



o 


Ton/iHje 


X 


© 


o 


pacTpecmHje 




© 


o 


BJiaxtHMje 


X 


© 


o 


Matbe 3ara?)eHO 


X 






(no response) 


X 


© 



Item parameters 



CTT 




IR r 






r-value 


p-value 






a 


b 


annual test 


0.38 


0.49 




annual test 


1.0 


0.1 


trial test 


0.34 


0.62 




trial test 


0.9 


-0.6 



Analysis of alternatives 



Response 

code 


Number of 
responses 


p-value of 
alternative 


r-value of 
alternative 


Median response 
time fs] 


© 


80 


0.09 


-0.17 


44 


© 


549 


0.62 


0.34 


40 


© 


66 


0.07 


-0.09 


44 


© 


178 


0.20 


-0.19 


44 




15 


0.02 


-0.13 


46 



PP013009 




Ability 
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Question #25 



PP083095 



TpM KaniMKe co/iM cunaHe cy y nocyfly ca boaom. Bofla y Kojoj je pacmopeHa co 
CTaB/beHa je y wepny fla npoKyBa. Bofla ca corny je McnapaBa/ia CBe aok Ha flHy 
niepne HMje ocra/ia caMO co. Lilia je MCTpaxtHBan xieo fla noKa>Ke obmm 
omeflOM? 



Ofla6epn jeflaH oflroBop 



O 

o 

o 

o 



fla BOfla McnapaBa Kafla ce jaxo 3arpeje 
fla ce co pacTBapa y boam 

fla je pacTBapai-be co jim y boah noBpaTHa npoMeHa 
fla je pacTBapai-be co nw y boam HenoBpaTHa npoMeHa 
(no response) 



x © 

X © 

✓ d> 

X @ 
X ® 



Item parameters 



CTT 




IR] 






r-value 


p-value 






a 


b 


annual test 


0.23 


0.20 




annual test 


0.7 


2.1 


trial test 


0.21 


0.29 




trial test 


0.6 


1.8 



Analysis of alternatives 



Response 

code 


Number of 
responses 


p-value of 
alternative 


r-value of 
alternative 


Median response 
time [s] 


© 


265 


0.30 


0.08 


82 


© 


217 


0.24 


-0.20 


73 


© 


256 


0.29 


0.21 


81 


© 


131 


0.15 


-0.09 


90 




19 


0.02 


-0.08 


71 



PP083095 




Ability 
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Question #26 



PK101118 



Ako ch OKpeHyT jimieM npeMa M3/iacKy cyHua, Koja je CTpaHa CBeTa M3a 
Te6e? 

Ofla6epw jeflaH oflroBop. 



o 


MCTOK 


X 


© 


o 


3ana,q 


V 


© 


o 


ceBep 


X 


© 


o 


jyr 


X 


@ 


(no response) 


X 


© 



Item parameters 



CTT 




IR r 






r-value 


p-value 






a 


b 


annual test 


0.29 


0.55 




annual test 


0.7 


-0.3 


trial test 


0.30 


0.58 




trial test 


0.8 


-0.5 



Analysis of alternatives 



Response 

code 


Number of 
responses 


p-value of 
alternative 


r-value of 
alternative 


Median response 
time [s] 


© 


83 


0.09 


-0.09 


39 


© 


517 


0.58 


0.29 


37 


© 


78 


0.09 


-0.12 


37 


© 


195 


0.22 


-0.19 


41 


<D 


15 


0.02 


-0.04 


27 



PK101118 
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Question #27 



PK111132 




Koje BpeMe noi<a3yje hscobhhk? 
Ofla6epn jeflaH oflroBop. 



o 


nona flBaHaecT 


X 


© 


o 


nona jeflaH 




© 


o 


neT MMHyTa ao mecT nacoBa 


X 


© 


o 


rnecT nacoBa h neT Mi/myTa 


X 






(no response) 


X 


© 



Item parameters 



CTT 




IR r 






r-value 


p-value 






a 


b 


annual test 


0.22 


0.90 




annual test 


0.8 


-3.3 


trial test 


0.21 


0.93 




trial test 


1.0 


-3.1 



Analysis of alternatives 



Response 

code 


Number of 
responses 


p-value of 
alternative 


r-value of 
alternative 


Median response 
time [s] 


0 


18 


0.02 


-0.10 


42 


© 


823 


0.93 


0.21 


31 


© 


3 


0.00 


-0.07 


59 


© 


34 


0.04 


-0.14 


44 




10 


0.01 


-0.10 


80 



PK111132 




Ability 



40 




Question #28 



PK121139 



npBa cpncKa .qpwaBa 6nna je: 
Ofla6epn jeflaH oflroBop. 



o 


KHeweBMHa Cp6nja 


X 


© 


o 


flywaHOBO uapcTBO 


X 


© 


o 


Be/iMKa Cp6nja 


X 


© 


o 


PamKa 








(no response) 


X 


© 



Item parameters 



CTT 




IR r 






r-value 


p-value 






a 


b 


annual test 


0.16 


0.52 




annual test 


0.4 


-0.2 


trial test 


0.27 


0.81 




trial test 


0.9 


-1.9 



Analysis of alternatives 



Response 

code 


Number of 
responses 


p-value of 
alternative 


r-value of 
alternative 


Median response 
time [s] 


© 


73 


0.08 


-0.11 


31 


© 


53 


0.06 


-0.13 


34 


© 


36 


0.04 


-0.18 


38 


© 


718 


0.81 


0.27 


20 




8 


0.01 


-0.12 


37 



PK121139 




Ability 
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Question #29 



PD143165 



Ca KojoM ofl HaBefleHMx flpxaBa ce rpaHMHM Peny6/im<a Cp6nja? 
Ofla6epw jeflaH oflroBop 



o 


MaT)apcKa 




© 


o 


TpHKa 


X 


© 


o 


CnoBem/ija 


X 


© 


o 


HeMaHKa 


X 




(no response) 


X 


© 



Item parameters 



CTT 




IR r 






r-value 


p-value 






a 


b 


annual test 


0.40 


0.86 




annual test 


1.4 


-1.8 


trial test 


0.24 


0.88 




trial test 


0.9 


-2.5 



Analysis of alternatives 



Response 

code 


Number of 
responses 


p-value of 
alternative 


r-value of 
alternative 


Median response 
time fs] 


© 


781 


0.88 


0.24 


24 


© 


20 


0.02 


-0.13 


35 


© 


60 


0.07 


-0.14 


34 


© 


19 


0.02 


-0.14 


35 




8 


0.01 


-0.04 


27 



PD143165 




Ability 
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Question #30 



PD153176 



IllTa ce HajHewfie npon3BOfln y paBHi/NapcKMM KpajeBMMa? 
Ofla6epw jeflaH oflroBop. 



o 


ayTOMo6n/in 


X 


© 


o 


xpaHa 




© 


o 


6aKap 


X 


© 


o 


HaMeinTaj 


X 




(no 


response) 


X 


© 



Item parameters 



CTT 




IR r 






r-value 


p-value 






a 


b 


annual test 


0.30 


0.68 




annual test 


0.8 


-1.2 


trial test 


0.19 


0.83 




trial test 


0.7 


-2.5 



Analysis of alternatives 



Response 

code 


Number of 
responses 


p-value of 
alternative 


r-value of 
alternative 


Median response 
time [s] 


© 


47 


0.05 


-0.05 


39 


© 


732 


0.82 


0.20 


28 


© 


72 


0.08 


-0.15 


35 


© 


23 


0.026 


-0.09 


39 




14 


0.02 


-0.05 


21 



PD153176 




Ability 



43 




Question #31 



PP021015 



no neMy ce 6n/bi<e pa3/im<yjy oa ocra/ikix >kmbmx 6nlia? 
Ofla6epw jeflaH oflroBop. 



o 


flHiny 


X 


© 


o 


pacTy m pa3BMjajy ce 


X 


© 


o 


CTBapajy xpaHy 




© 


o 


ocraB/bajy iiotomctbo 


X 






(no response) 


X 


© 



Item parameters 



CTT 




IR r 






r-value 


p-value 






a 


b 


annual test 


0.45 


0.63 




annual test 


1.3 


-0.6 


trial test 


0.39 


0.78 




trial test 


1.3 


-1.3 



Analysis of alternatives 



Response 

code 


Number of 
responses 


p-value of 
alternative 


r-value of 
alternative 


Median response 
time fs] 


© 


20 


0.02 


-0.14 


46 


© 


92 


0.10 


-0.29 


40 


© 


693 


0.78 


0.39 


36 


© 


74 


0.08 


-0.14 


47 




9 


0.01 


-0.13 


43 



PP021015 




Ability 



44 




Question #32 



PK093106 





Kom paflHMKy fie 6 mtm Haj/iaKine aa Byie TepeT? 
Ofla6epn jeflaH oflroBop. 




O a 


X 


© 


O 6 


X 


© 


B 


X 


© 


O r 






(no response) 


X 


© 



Item parameters 



CTT 




IR r 






r-value 


p-value 






a 


b 


annual test 


0.36 


0.80 




annual test 


1.0 


-1.6 


trial test 


0.23 


0.90 




trial test 


0.9 


-2.7 



Analysis of alternatives 



Response 

code 


Number of 
responses 


p-value of 
alternative 


r-value of 
alternative 


Median response 
time [s] 


© 


14 


0.02 


-0.10 


52 


© 


20 


0.02 


-0.11 


37 


© 


47 


0.05 


-0.13 


48 


© 


796 


0.90 


0.23 


39 




11 


0.01 


-0.12 


32 



PK093106 
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Appendix 2: List of elementary schools that participated 
in on-line testing 



1 Bratstvo, Aradac 

2 Jelena Cetkovic, Beograd 

3 Jovan Kursula, Varvarin 

4 4. oktobar, Vojvoda Stepa 

5 Dositej Obradovic, Vrba 

6 Bratstvo Jedinstvo, Vrbas 

7 Petar Petrovic Njegos, Vrbas 

8 Svetozar Miletic, Vrbas 

9 Desanka Maksimovic, Gornji 

Milanovac 

10 Hristo Botev, Dimitrovgrad 

11 Dr Aleksandar Sabovljev, Ecka 

12 Mosa Pijade, Zagubica 

13 Desanka Maksimovic, Zajecar 

14 2. oktobar, Zrenjanin 

15 Petar Petrovic Njegos, Zrenjanin 

16 Milinko Kusic, Ivanjica 

17 Fejes Klara, Kikinda 

18 Dura Jaksic, Kovin 

19 Dura Jaksic, Kragujevac 

20 Jovan Popovic, Kragujevac 

21 Moma Stanojlovic, Kragujevac 

22 Nada Popovic, Krusevac 

23 22. jul, Krcedin 

24 Dobrosav Radosavljevic Narod, 

Macvanska Mitrovica 

25 Vuk Karadzic, Negotin 

26 Braca Stefanovic, Neuzina 

27 Branko Miljkovic, Nis 



28 Bubanjski heroji, Nis 

29 Vozd Karadorde, Nis 

30 Miroslav Antic, Nis 

31 Mirko Tomic, Obrez 

32 Dura Jaksic, Oreskovica 

33 Miroslav Antic, Palic 

34 Jovan Jovanovic Zmaj, Pancevo 

35 Dura Jaksic, Ravni 

36 Ivo Lola Ribar,Ruma 

37 Backa Palanka, Sveti Sava 

38 Branko Radicevic, Smederevo 

39 Dimitrije Davidovic, Smederevo 

40 Ivo Lola Ribar, Sombor 

41 Vuk Karadzic, Srbobran 

42 Dura Jaksic, Srpska Crnja 

43 Vuk St. Karadzic, Starcevo 

44 Stefan Nemanja, Studenica 

45 10. oktobar, Subotica 

46 Ivan Goran Kovacic, Subotica 

47 Majsanski put, Subotica 

48 Secenji Istvan, Subotica 

49 Matija Gubec, Tavankut 

50 Aleksa Dejovic, Temerin 

51 Mladost, Tomasevac 

52 Petar Drapsin, Turija 

53 Slobodan Sekulic, Uzice 

54 Milica Pavlovic, Cacak 

55 Sveti Sava, Citluk 

56 Laza K. Lazarevic, Sabac 
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Appendix 3: Trial test sample characteristics 

The trial test was conducted on convenience sample that consisted of 926 
students from 50 elementary schools. All schools interested in on-line testing 
took part in the study. They were allowed to select students for the testing 
according to their own needs. In some schools teachers intended to test all 
fourth-grade students; on the other hand, but also there were schools where 
teachers were just interested to see what is on-line testing like and how does it 
work. Schools that applied for the testing surely belong to better organized 
schools where computers in computer room have Internet connection and where 
teachers show interest to participate in the project of this kind. 

Considering the data obtained from schools about the way selection of students 
was done, we can see that 50% of the sample consists of more or less randomly 
selected students (random selection, all students from a class, first 15 students 
from the register, etc.) while 40% of students were selected according to school 
marks, computer skills and motivation, or because students had asked to 
participate in the testing. For 10% of students schools did not provide data about 
the way of selection. 

If we compare ability estimations 2 for each of these groups, we can see that 
students selected for the trial test performed significantly better than students at 
the annual test. Mean abilities for each of these groups are given in Table 4 with 
the estimated standard error is given in parenthesis. 





Number of 
students 


Mean ability 


Representative sample (the annual test] 

Convenience sample (the trial test] 

• random sample within a class or all students in a class 

• students selected according to school marks, 
motivation and similar 

• without information about the way of selection 


1846 

903 

• 454 

• 354 

• 95 


0.00(2] 

0.61(3] 

0.52(4] 

0.74(5] 

0.62(8] 



Table 4: Mean ability for students selected in different way 



In order to see how convenience sample is different from the representative one, 
we have compared students’ grades in Nature & Society at the end of previous 
semester. We can notice that students from the convenience sample have much 
greater mean course grade (Table 5). 





Number of 
students 


Mean course 
grade 


Representative sample (the annual test] 

Convenience sample (the trial test] 

• random sample within a class or all students in a class 

• students selected according to school marks, 
motivation and similar 

• without information about the way of selection 


4115 

903 

• 454 

• 354 

• 95 


4.10(2] 3 

4.54(3] 

4.31(5] 

4.76(3] 

4.82(5] 



Table 5: Mean course grade in Nature & Society for students selected in different way 



2 For the reasons of comparison on the same scale with annual test, mean estimated ability is 
displayed on the scale of the annual test. 

3 This data is obtained from questionnaire for the preparation of standard setting test for the 
same population. 
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Appendix 4: Questionnaire for students 

After the testing was finished, teachers asked students to fulfill short on-line 
questionnaire before they log out. This questionnaire had three questions 
concerning course grade in Nature & Society, the frequency of computer usage, 
and applied effort. The questionnaire was answered by 60% of students who 
took part in the testing. 

• Question number 1: What grade did you have in Nature & Society at the 
end of previous semester? 

(proposed answers from 1 (lowest) to 5 (highest)) 

We can see that most of students had grade 5 (highest) in Nature & Society. The 
graph of estimated ability depending on the grade at the end of semester is given 
on the Figure 6. For the sake of comparison, we have denoted girls' results with 
red dots, and boys' results with blue. Obviously, students with higher course 
grade, in average, had better test results. 



Grade 


Number of 
students 


1 


2 


2 


21 


3 


40 


4 


94 


5 


375 


unknown 


371 




48 



• Question number 2: How often do you use computer? 

1 this is my first time to use computer 

2 almost never 

3 approximately once a month 

4 several times a month 

5 almost every day 

From answers to this question, we can see that great majority of students use 
computer regularly, and that more than a half of them use computer almost 
every day. Finding from our previous study in 2007 [4] that students who use 
computer more frequently have better results is not so obvious any more. The 
graph of estimated ability depending on the frequency of computer usage is 
given on Figure 7. We have denoted that girls' results with red and boys' results 
with blue dots. Here we can see that there is no significant difference between 
how often boys and girls use computer. 
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usage 
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33 
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22 


4 


125 


5 


342 


unknown 


373 
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• Question number 3: How much effort did you apply to answer all 
question in the test? 

(proposed answers from 1 (not at all) to 5 (very much)) 

From the answers to the third question, we can see that more than a half of 
students said they applied great effort to do this test. The graph of estimated 
ability depending on applied effort is given on Figure 8. We have denoted girls' 
results with red and boys' results with blue dots. We can see that students who 
applied greater effort achieved better result at the test. Also, we can see there is 
no significant difference between effort applied by boys and girls. 
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2 
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25 


4 


107 
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394 


unknown 


375 
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